DISTRIBUTIVE LATTICES, AFFINE SEMIGROUPS, AND 
BRANCHING RULES OF THE CLASSICAL GROUPS 



SANGJIB KIM 

Abstract. We study algebras encoding stable range branching rules for the pairs 
of complex classical groups of the same type in the context of toric degenerations of 
spherical varieties. By lifting afiine semigroup algebras constructed from combinatorial 
data of branching multiplicities, we obtain algebras having highest weight vectors in 
multiplicity spaces as their standard monomial type bases. In particular, we identify a 
family of distributive lattices and their associated Hibi algebras which can uniformly 
describe the stable range branching algebras for all the pairs we consider. 



1. Introduction 

1.1. Let us consider a pair of complex algebraic groups G and H with embedding H C G 
and their completely reducible representations Vq and Vh- If Vh is irreducible, then a 
description of the multiplicity of Vh in Vg regarded as a representation of H by restriction 
is called a branching rule for (G,H). By Schur's lemma, the branching multiplicity is 
equal to the dimension of the space HomH(VH,VG), which we will call the multiplicity 
space. 

1.2. In this paper, we shall consider branching rules of the polynomial representations 
of the following pairs (G,H) of complex classical groups: (GLra, GL^), (Sp2ra) Sp2n)i 
(S0p,S0q). Our goal is to study branching rules for (G,H) collectively in the context 
of toric degenerations of spherical varieties and to obtain an explicit description of the 
multiplicity space HomH(V^, Vq) when the length €(A) of highest weight A for G satisfies 
the following stable range condition: 

(1) £(A) <mfor (GU,GLn); 

(2) £(A) <nfor [Svz^.SvzJ, (SOzm, SOin+i ), (SOzm+i , SOin+i ); 

(3) £(A) < n for (S02^,S02n), (SOznx+l , S02n). 

We shall construct an algebra whose graded components are spanned by the highest 
weight vectors of irreducible representations of H appearing in each irreducible repre- 
sentation of G. 

1.3. To give a slightly more detailed overview, let us consider the ring Tq of regular 
functions over G/Ug where \1q is a maximal unipotent subgroup of G. This ring is called 
the flag algebra for G, because it can be realized as the multi-homogeneous coordinate 
ring of the flag variety. As a G-module, the flag algebra Tq contains exactly one copy 
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of every irreducible representation of G |LT791 ILT85] . and in this context the author 
studied polynomial models for Tq and their fiat degenerations |Ki081 [Ki09| . 

By highest weight theory, the llH-invariant subspace of consists of the highest 
weight vectors of irreducible representations of H appearing in Vg . Therefore, the Uh- 
invariant subalgebra of leads us to study the branching rules for (G, H] collectively: 

(1.3.1) J-,^" = Z(V^)''" 

AeG 

AeG neH 

where m(V[J, V^) is the multiplicity of in Vq. 

Moreover, we can impose a graded structure on J^^" so that its graded components 
correspond to the multiplicity spaces: 

m(V|i,V^) (V|;)^" =HomH(v;;,v^) 

for (A, G G X H. In this sense, we may call the branching algebra for (G,H). 
This algebra was introduced by Zelobenko. See |Ze62| and |Ze73| . 

1.4. Recently, Howe and his collaborators studied branching algebras for classical sym- 
metric pairs, especially their toric degenerations and expressions of branching multi- 
plicities in terms of Littlewood- Richardson coefficients |HJLTW| IHTW] . In the cases 
this paper concerns, using known combinatorics of branching rules, we can explicitly 
describe the multiplicity spaces and their degenerations. More specifically, we show that 
the stable range branching algebras are deformations of semigroup algebras of generalized 
semistandard tableaux or equivalently Gelfand-T^etlin patterns, and therefore provide a 
precise connection between the multiplicity space and the combinatorial objects which 
count its dimension. 

Starting from combinatorial data of stable range branching multiplicities, we shall 
construct an affine semigroup and its semigroup algebra graded by the pairs of highest 
weights for the classical groups G and H listed in §1.21 This algebra can be realized as 
a Hibi algebra over a distributive lattice. Then, by using toric deformation techniques, 
we lift the Hibi algebra to construct a polynomial model of the branching algebra for 
(G,H). We study its finite presentation and standard monomial type basis. It turns out 
that there is a particular type of distributive lattices whose Hibi algebras can uniformly 
describe stable range branching algebras for all the pairs (G,H) we consider. 

We remark that this Hibi algebra structure in branching problems has interesting 
counterparts in tensor product decomposition problems, which can be explained by reci- 
procity properties between branchings and tensor products in representation theory. For 
this direction, we refer readers to |HL07t [HKH IKL) . 

1.5. This paper is arranged as follows: In Section 2, we develop the combinatorial tools 
we will use. In Section 3, we study the branching algebra for (GLm, GLn) and its toric 
degeneration. In Section 4 and Section 5, we study the distributive lattices and affine 
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semigroups associated with the branching rules for (Sp2ttt,, Sp2n) and (S0p,S0q), and 
construct the corresponding stable range branching algebras. 

2. Combinatorics of Branchings 

This section is to prepare us the combinatorial ingredients we will use to construct 
stable range branching algebras. 

2.1. The Gelfand-Tsetlin{GT) poset for GL^, is the poset 

= jxj'' : 1 < i < m, 1 < j < i} 

satisfying x^^^' > xj''' > x|^^' for all i and j. The elements of V^n, can be listed in 

a reversed triangular array so that xj''' are weakly decreasing from left to right along 
diagonals as GT patterns are originally drawn |GT50| . Counting from bottom to top, 
we will call x'^' = (x|^',X2^\ • • • ,x[^') the r-th row of V-m- 

Definition 2.1.1. (1) For m > n, the GT poset for (GLra, GL^) is the following 
subposet of Fra-' 

= {^f G : n < i < m} . 
(2) In r^, for vr\.>k we define the GT poset of length k as 

C,k = {xf' Gr-:j<k}. 
For example, Fg^ can be drawn as 

AS] ^(6) (6) (6) 

1 2 3 4 

^{5] (5) (5) (5) 

('9 1 1 \ ^1 ^2 ^3 ^^4 

y^-^-^> ^(4) (4) (4) (4) 

1 2 3 4 

^(3) (3) (3) 

X, X.3 

2.2. Next, let us consider the set Cm, of all nonempty subsets of {1 , 2, • • • , m}. We shall 
write 

I = [i-l , • • • , iJ 

for the subset consisting of 1^ with 1 < ii < • • • < ta < m. The length |I| = a of I is the 
number of elements in I. 

The following partial order ^, called the tableau order, can be imposed on Crn- for 
two elements I and J of /I^a, we say I ^ J, if [I| > |Jt and the c-th smallest element in I 
is less than or equal to the c-th smallest element in J for 1 < c < |J|. Then, with 
^ forms a distributive lattice whose meet A and join V are, for I = [li, • • • ,ia\ and 
J = • • • ,jb] with a < b, 

lAJ = [min(ii,ji),--- ,min(ta,jQ),lQ+i,--- ,ib] 
IVJ = [max(li,ji),--- ,max(ta,ja)]- 
It is straightforward to check the following subposets are also distributive lattices. 
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Definition 2.2.1. (1) For m > n, the distributive lattice for (GLt^i, GL^) is 
the subposet of consisting of the following elements: 

[1,2,--- ,r-l,r,ai,a2,--- ,0.$], 
[1,2,--- ,r-l,r], 
[ai, a2, ■ ■ ■ , as] 

where r < n and n + 1 < a-\ < • • • < as < m. 
(2) For k < m, we let ^ denote the subposet of consisting of elements of 
length not greater than k; 

C^,^ = {l€Cl:\l\<]c]. 

2.3. The poset structure of can be read from the GT poset V^^. of length k. For 
this, let us impose a partial order on the set of order increasing subsets of as follows. 
For two order increasing subsets A and B of P^^, we say A is bigger than B, if A C B 
as sets. Note that here we use the reverse inclusion order on sets, because we use order 
increasing sets instead of order decreasing sets. 

Proposition 2.3.1. There is an order isomorphism between C^^^ and the set of 
order increasing subsets ofV^^^. 

This is an easy computation similar to [Ki081 Theorem 3.8]. For each I G 
define the corresponding order increasing subset Aj of ^ as 

(2.3.1) Ai= U {xfU™...,x™} 

rL<i<m 

where St is the number of entries in I less than or equal to i. For example, the subset of 
4 given in ()2.1.ip corresponding to I = [1,4, 6] S /^g4 is 



X2 

(4) ,f4) 



(3) 

x; 

Then, it is straightforward to check that this correspondence gives an order isomorphism. 
In fact, this Proposition gives an example of Birkhoff 's representation theorem or the 
fundamental theorem for finite distributive lattices |Sta971 Theorem 3.4.1]. See [KiOSl 
§3.3] for further details. 

For k < n and d > 0, we can identify F^ ^ with F^^^ ^ by shifting the i-th row x'^^ up 
to the (i + d)-th row x''-^'^' for n < i < m, and then the above Proposition gives 

Corollary 2.3.2. For k < n and d > 0, there is an order isomorphism between 
distributive lattices 

^m,k — ■'-Ta+d,k 
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2.4. A shape or Young diagram is a left-justified array of boxes with weakly decreasing 
row lengths. We identify a shape with its sequence of row lengths D = (rj , r2, • • • )• The 
following example shows the shape D = (4,2,1): 



If I is maximal with ^ 0, then we call I the length of D and write £(D) = I. If we 
flip a shape D over its main diagonal that slants down from upper left to lower right, 
then we obtain its conjugate D^. With the previous example, we have £(D) = 3 and 
= (4,2, l)t = (3,2,1,1). For F = (fi,f2,---) and D = (di,d2,---), if fr > dr for all 
r, then we write F 2 D and let F/D denote the skew shape having F as its outer shape 
and D as its inner shape. 

2.5. Consider a multiset {Ii, • • • , Is} C Cm. with jld = Ic for each c. A concatenation t 
of its elements is called a tableau, if they are arranged so that Ic > Ic+i for all c. The 
shape sh(t) of t is the Young diagram (li , • • • , 1$)^ and the length £(t) of t is the length 
of its shape. If {Ii , • • • , IJ is taken from the subposet C^, then we shall specify the outer 
and inner shapes of t. 

Definition 2.5.1. A standard tableau t for (GLm, GL^) is a multiple chain 

t=(Il ^•••^Is) 

in C^. The shape shn(t) oft is F/D where 

F = (|Ii|,--- ,|Is|)^ andD = (di,--- ,dn) 
and dr is the number of r's in t for 1 < r < n. 

For example, the multiple chain [1,2,3,6] ^ [1,2,5,6] ^ [1,2,6] ^ [1,4] ^ [5] ^ [5] in 
£^4 forms a standard tableau for (GL6,GL3) of shape (6,4,3,2)/(4,3, 1): 



1 


1 


1 


1 


5 5 


2 


2 


2 


4 




3 


5 


6 






6 


6 









which can be, after erasing r < 3, identified with the following skew semistandard tableau 











5 


5 








4 








5 


6 








6 


6 











2.6. The following set of pairs of Young diagrams will be used frequently: for a > b, 

Aa,b = {(F, D) : £(F) < a, 1{D) < b, F D D}. 

We note that if (F, D) e Aa^b, then £(D) < min(£(F),b). This is because FDD implies 
£(F)>£(D). 
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2.7. Let 7^(F, D) denote the set of all standard tableaux for (GLt^i, GL^t^) whose shapes 
are F/D. For each k with n < k < ra, we consider the following disjoint union over A]^^^ 

(F,D)6Ak,n 

As illustrated by the example in §2.5t if we identify the elements of with single- 
column tableaux, then our definition of standard tableaux for (GL^a, GLn) of shape F/D 
agrees with the usual definition of skew semistandard Young tableaux of shape F/D with 
entries from {n + 1 , • • • , m}. 

By setting tableaux in the context of a finite distributive lattice (Definition 12.5. ip . we 
can exploit an additional structure: Proposition 12.3.11 leads us to study C^^. in terms of 
the order increasing subsets of V^^, and the order increasing subsets of give rise to 
the order preserving maps from ^ to {0, 1 }. More generally, 

Definition 2.7.1. A GT pattern for (GLtt^, GLn) is an order preserving map from 
the GT poset F^ for (GL^n, GLn) to the set of non-negative integers: 

p : F^ ^ Z>o. 

The r-th row of p is (pfxj'^^), • • • , p(x[^')) for n < r < m. The type of p is F/D where 
F and D are its m-th row and the n-th row respectively. 

Note that if £(F) < k, then the support of every GT pattern p of type F/D lies in the 
GT poset F^ ^ of length k. Therefore, we have GT patterns defined on F^ ^ 

Let VJ^iV, D) denote the set of all GT patterns for (GLm., GLn) whose type is F/D. Then 
for each k with n < k < ra, we consider the following disjoint union over A]^y. 

(2.T.1) P^,,k= U ^m(f,D) 

(F,D)GAk,n 

2.8. Since the sum of two order preserving maps is an order preserving map, I^^y 

a semigroup with function addition as its multiplication, or more precisely a monoid 
with the zero function as its identity. We further note that P^]^ is generated by the 
order preserving maps from F^^ to {0,1}. Then, by identifying each GT pattern p 
with (p(x|^')) € l}^ where N is the number of elements in F^^, we see that V^^^ can 
be understood as an affine semigroup, i.e., a finitely generated semigroup which is 
isomorphic to a subsemigroup of containing for some N [BH93) . 

This semigroup structure on GT patterns provides a simple bijection between 
and Vl^^. 

Proposition 2.8.1. For each (F, D) € A^^^, there is a bijection between 7^(F, D) 
andr^[¥,D). 

Proof. The bijection in Proposition 12.3.11 provides the bijection between and the 
set of characteristic functions on order increasing subsets of F^. This bijection can be 
extended to multiple chains in as follows. Let t = (Ii ^ • • • ^ Ic) be a multiple chain 
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in and pi^ be the characteristic function on the order increasing set Aj^ corresponding 
to It- given in ()2.3.ip for each r. Then we can consider the following correspondence: 

c 

(2.8.1) t=(Ii ^•••^Ic)^Pt = ^PI,. 

r=1 

Since the order preserving characteristic functions over generate V^, this corre- 
spondence gives a bijection between 7^(F, D) and ^^(F, D). For further details, see 
[How05[[Kl08| . □ 

2.9. We remark that by identifying GT patterns p with their images (p(x^')), our 
definition is equivalent to the usual definition of GT patterns. The correspondence 
between the set of semistandard tableaux and the set of GT patterns given in the above 
proposition is the same as the known bijection or conversion procedure (e.g., |GW091 
§8.1.2]), which is usually explained by successive applications of the Pieri's rules. 

For example, a pattern p € 7^14 can be visualized by listing its value at xj''' € Fg^ 

3 3 3 1 

3 3 2 
(2.9.1) .... 
^ ' 3 2 10 

2 2 1 

Then it is the sum of the GT patterns 

1111 1110 1110 
1110 1110 1100 
1110^ 1100^ 1000 
111 1 1 

corresponding to the elements [1,2,3,6] ■< [1,2,5] ^ [4,5,6] of £g4. This multiple chain 
can be identified with the following standard tableau in T^^ 



1 


1 


4 


2 


2 


5 


3 


5 


6 


6 





(2.9.2) 

of shape (3,3,3, l]/(2,2, 1). Note that to ()2.9.2p . we can apply the usual conversion 
procedure (e.g., |GW091 §8.1.2]) to obtain its corresponding pattern — by successively 
striking out the boxes with 6, 5, and 4 in the tableau ()2.9.2p . we obtain each row of the 
pattern dXgi]) . 

2.10. Now we study an algebra attached to C^^^ in terms of the set of standard 
tableaux and the set ^ of GT patterns. 

Definition 2.10.1 ( [Hi87] ). The Hibi algebra T-L[l-] over a finite distributive lattice 
L is the quotient ring of the polynomial ring C[2:^ : y G L] by the ideal generated by 
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2^a2^(3 "Z^ApZccvp for all incomparable pairs (a, (3) o/L; 

n[L) = C[Zy : y G L]/(ZaZ|3 - z^ApZavp) 
Let us consider the Hibi algebra over ^ 

We shall identify the monomials Yir ^ir k ^ith the tableaux consisting of elements 

It G ^mk- -^^^ example, the above tableau (12.9.2P will be used to denote the monomial 

2^[1236]2^[125]Z[456] S 7114 

Recall that standard tableaux are multiple chains in (Definition I2.5.ip . Then the 
following is from a general property of Hibi algebras |Hi87t IHowOSj . 

Lemma 2.10.2. (1) The set T^i^ of all standard tableaux for (GLtu, GL^) whose 
shapes are F/D with £(F) < k form a C-basis for the Hibi algebra T-L^^- 
(2) In particular, Ti^^ is graded by A]^^^, and the set 7^(F, D) of standard 
tableaux for (GLtu, GLn) of shape F/D form a C-basis for the (F, D) -graded 
component of V.^^. 

It is shown in |Ki08t Corollary 3.14] that the Hibi algebra over is isomorphic to 
the semigroup algebra of GT patterns defined on Vm- This fact combined with the above 
Lemma leads us to study the Hibi algebra T-L^i^ over in terms of the semigroup 

algebra C [V^ ^] of the affine semigroup ^ given in (I2.7.ip . 

Note that for pi and pz € of types Fi/Di and F2/D2 respectively, the type of 

(Pi + P2) is (Fi + F2)/(Di + D2), and therefore €[7-"^^,^ J is graded by pairs of shapes 

(F,D)eAk,n 

where C[7'^](i:_o) is the space spanned by ^^(F, D]. 

Proposition 2.10.3. (1) The Hibi algebra T-L^^ over is isomorphic to the 
semigroup algebra CiV^^] of the GT patterns for (GLra, GLn). 
(2) The set V^^^iV, D) of GT patterns for (GL^x, GLn) of type F/D is a C-basis for 
the (F, D) -graded component C['PJJl](f,d) • 

Proof. For the first statement, by Proposition 12.3.11 there is a bijection between the set 
of elements xj € "HJ^^ ^ ^ ^mk characteristic functions over order increasing 
subsets of r^^. This gives an algebra isomorphism from T-L^i^ to C['P^|^] via ()2.8.ip . See 
[HowOSl IKiOS] for further details. The second statement follows from Proposition 12.8.11 
and the above Lemma. □ 



3. Branching Algebras for (GLra, GLn) 

In this section, our goal is to construct an algebra encoding branching rules for 
(GLm, GLn) and study its toric degeneration. For later use, we will construct a fam- 
ily of algebras parametrized by the length k of highest weights for GL^. 
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3.1. Recall that the set of Young diagrams F with £(F) < m can be used as a labeling 
system of irreducible polynomial representations of GL^n by identifying dominant weights 
(fl > ••• > fm) G ^>o of GLtt, with Young diagrams (cf. |GW09[ §3.1.4]). We let pj^ 
denote the irreducible polynomial representation of GL^-n, labeled by Young diagram F. 

Then the branching algebra for (GLta, GL^) will be graded by the set Aitl^i 

de- 
fined in §2.61 and its graded components will correspond to the multiplicity spaces 

HomGLn(Pn,Pm) (F,D) € Anx,n- 

3.2. For Young diagrams F = (fi , f2, • • • ) and D = {d-\, dj, ■ ■ ■ ), "we write 

F □ D 

if fr > dr > fr+1 for all r, and say F interlaces D. 

Proposition 3.2.1. (1) For Young diagrams F and D with £(F) < m and ^{D) < 
m — 1, the multiplicity of pj^_i in pj^ is 1 i/F □ D, and otherwise. 
(2) The number of GT patterns in V^iV, D) is equal to the multiplicity m(p{^, pj^) 
of p° m pj^. 

Proof. The first statement is known as the Pieri's rule (see, e.g., |GW091 §8.1.1]). The 
second statement can be obtained by applying the Pieri's rule for (GLi^+i,GLk) succes- 
sively for n < k < m — 1 . The multiplicity of p{^ in pj^ is equal to the number of the set 
{(Em-l , ■■■ , tn+i )} such that 

F □ E^_i □ E^_2 □ • • • □ En+i □ D. 

Note that by setting F = and D = En, Ei represents the i-th row of a GT pattern in 
V^[¥,D) forn < i < m. □ 

Prom Proposition 13.2.11 and Proposition 12.8. 1[ we have 

Corollary 3.2.2. For (F, D) € Am,n, the branching multiplicity m(p{^,p|^) is equal 
to the number of standard tableaux for (GLra, GLn) whose shapes are F/D. 

3.3. To construct a family of branching algebras for (GLtu, GLn) parameterized by the 
length k, let us review a polynomial model for the flag algebra. We assume m > k and 
let GLra X G Lie be acting on the space Mra,k = (g) C'^ of m x k complex matrices by 

(3.3.1) (91,92) •Q = (gl)"^Qg2' 

for gi € GLra, 92 £ GLi^, and Q € M-m.,Y- Then under the GLra x GLi^ action, the 
coordinate ring C[Mra,k] of Mra,k has the following decomposition: 

C[Mn^,k] = Y- Pm ® Pk 

f{F]<k 

where the summation is over F with length not more than k. This result is known 
as G Era" G Lie duality (e.g., |GW09[ IHow95| ). If is the subgroup of GL^ consisting 
of upper triangular matrices with 1 's on the diagonal, then by taking 11)^ = 1 x 11)^ 
invariants, we have 

C[Mm,k]^^= Y. Pm^tPk)""^ 
e(F)<k 
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3.4. This representation decomposition turns out to be compatible with the multiphca- 
tive structure of the algebra. Since the diagonal subgroup of GL]^ normalizes li^, 
C[Myn_^^]^^ is stable under the action of A]^. Note that by highest weight theory (e.g., 
|GW091 §3.2.1 and §12.1.3]), (p^)^'' is the one dimensional space spanned by a highest 
weight vector of p^, and Ai^ acts on (p^)^'' by the character 



(()F(diag(ai , • • • , ak)) = a/ • • • a 



given by Young diagram F = (fi,f2, • • • ,fk)- Thus, Pm — Pm ® (p^)^'' is the space of 
Ak-eigenvectors of weight cj)]: in C[Mm,k]^'' and the C-algebra C[Mm,k]^'' is graded by the 
semigroup of dominant polynomial weights for GLk, or equivalently the subsemigroup 
C A+ of dominant weights for GLm: 



(3.4.1) 



-'m 



/2 
-'m 



C 



P(F)<k 
rm 



where we identify (ri , • • • , Tk) G I^^q with (ri , • • • , tk, 0, • • • ,0) G Z>q. 

3.5. A finite presentation of C[Mm,k]^'' in terms of generators and relations is well 
known — all the Uk-invariant minors on Mt^^ k form a generating set and they satisfy the 
Pliicker relations. To explain more details, let us consider a subposet >Cm,k = ^mk °^ 
consisting of elements I = [ii,i2, • • • ,1^ such that |I| < k (cf. Definition 12.2. ip . 

For each Q € Mm,k, we let 6i(Q) denote the determinant of the submatrix of Q = (ta^b) 
obtained by taking the ii , 12, • • • , v-th rows and the 1 , 2, • • • , r-th columns: 



(3.5.1) 



5i(Q) = det 



till ti,2 

ti2l ti22 

tirl ti^2 



Definition 3.5.1. A product 8i^8i^---5i^ is called a standard monomial (or GLm 
standard monomial), if its indexes form a multiple chain t = (Ii ^ I2 ^ ■ ■ ■ ^ Ir) 
^m,k CLfT-d, write 

At = 5i,6i2 • • • 5i,. 

Then we define the shape of a standard monomial At to be the shape of t, i.e., 

(|Ill,|l2l,--- ,|Irir. 



Proposition 3.5.2 ( [GLOU pp. 233, 236]). (1) For I, J G £m,k, the product 5i6j G 
C[M 

m,k] ^ can be uniquely expressed as a linear combination of standard 
monomials 

(3.5.2) 5i6j = ^Cr5s,5T, 

r 

where, for each r with Cy ^ 0, St ^ T,- in £m,k CLT^d, SrUTr = lUj as sets. 
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(2) On the right hand side, Sjaj^ivj appears with coefficient 1, and ^ lAJ and 
I V J ^ Tr for all r with Cr 7^ 0. Moreover, for each (Sr, T,.) 7^ (I A J, I V J), let 
h be the smallest integer such that the sum s of the h-th entries of Sy and 
Tr is different from the sum so of the h-th entries of I and J. Then s > sq. 

By applying the straightening relations ()3.5.2p . we can find a C-basis for C[Mm.,k]^'^- 
The following is well known. See, for example, |BH93[ [DEP82| IGLQlj IHod43] . For this 
particular form, see [Ki08[ Theorem 4.5, Remark 4.6]. 

Proposition 3.5.3. Standard monomials associated with multiple chains t in 
-Cm,k form a C-basis for C[Mra,k]^''- More precisely, standard monomials At with 
sh(t) = F form a weight basis for the GLm irreducible representation pj^ C C[Mra,k]^'' 
with highest weight F. 

We specify the following properties of the standard monomial expression of 5i6j for 
I, J € ^ of length not more than k, which can be easily derived from the above 
Proposition. 

Corollary 3.5.4. Let I and J be incomparable elements in C^-^^ with |1| > |J|. Con- 
sider the standard monomial expression of the product 61 5 j given in l[3.5.^) . Let 
us denote the standard tableau Sy ^ Tr by ty. Then, for each r with nonzero Cr, 

(1) the shape shn(tr) is F/D where F = (|I|, |J|)^ and D = (di, di, • • • ) where d^ is 
the number of h 's in the disjoint union lU J for 1 < h, < n; 

(2) all the entries in the h-th row of are bigger than or equal to h for 1 < h, < 
min(rL, 

(3) if we denote the numbers of entries less than or equal to h in Sr and Tr by 
(Xh and (3h respectively, then an + |3h < 2h for 1 < h < mm(n, |I|). 

Example 3.5.5. For I = [1,2,5,6] and J = [1,3,4] from Cl^, we have 

S [1 256] S [1 34] = S [1 246] 5 [1 35] " ^ [1 236] ^ [1 45] 

+^[1235] S [146] — S[1245]S[136] " ^ [1 234] ^ [1 56] 

Note that shn,(tr) = (2,2,2, l)/(2, 1) for all the terms tr on the right hand side. 

3.6. Let m > n. To consider the branching rules for (GLra, GLn.), we use the following 
embedding of GLn, in GLra: for X G GLn, 

X 



I 



G GLr 



where I is the (m — n) x [m — n] identity matrix and O's are the zero matrices of proper 
sizes. 

From ()3.4.ip . by taking Un-invariants, we have 

(3.6.1) C[Mnr,k]^'^"^^ = {P'-^Y^ 

{(F)<k 

{(F)<k D 
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where m(pj^,pj^) is the multipUcity of appearing in pj^, and (p{^)^'^ is the one- 
dimensional space spanned by a highest weight vector of pj^ . 

Definition 3.6.1. For ra > k, the length k branching algebra for {GLyny GLn) is the 
(Un X U]^) -invariant ring o/C[Mn^^J 



3.7. Note that for I G -C^^^ all the minors 5i are invariant under the subgroup x Ui^ 
of GLn X GLi^ with respect to the action (I3.3.ip . In fact, the length k branching algebra 
for (GLra, GLn) is generated by {5i : I G ^C^^^J. 

Theorem 3.7.1. For each k with ra > k, the branching algebra for (GL^a) GLn) 
is graded by A^^^ 

and the standard monomials At for t G 7^(F, D) form a C-basis of the (F, D)-graded 
component i3J^]^(F, D). 

Proof. For I G C^^, the determinant functions 5i as elements of C[Mni,k]^'' satisfy the 
relations (j3.5.2p and by keeping track of the entries of I and J in this relation, we can easily 
see that all Sr and Tr appearing on the right hand side of ()3.5.2p are elements of C^-^ 
and sh,n(Sr ^ Tr)'s are the same for all r as in the first statement of Corollary 13.5.41 By 
applying these relations repeatedly, we can express every monomial in {6i : I G JC^ ^} as a 
linear combination of standard monomials of the same shape. In particular, the algebra 
is graded by the shapes shn(t) G Ai^^n of standard monomials for (GLm, GLn). Now, 
it is enough to show that for each shape F/D with (F, D) G Aj^ ni the number of standard 
monomials At for t G 7^(F, D) is equal to the multiplicity of pn in p|n, which is Corollary 
[3:231 □ 

Note that the standard monomials At for t G 7^(F, D) are invariant under the action 
of Un and scaled by the character (|)d under the action of the diagonal subgroup of GLn: 



(3.7.1) diag(ai,--- ,an) • At = 4)d (diag(ai, • • • , an)) At 

= (af'•••Q^)At 

for D = (di, • • • , dn). This shows that standard monomials At for t G 7J^(F, D) are the 
highest weight vectors of the copies of pn in p|^. Accordingly, we have 

Proposition 3.7.2. The standard monomials At with t G 7^(F, D), as C-basis el- 
ements of B^i^[¥.,D), are the highest weight vectors of the copies of pn in p{^. 
Therefore, we have 

S;;,k(F,D) =HomGLn(Pn,Pm)- 
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3.8. Toric degenerations of the branching algebras can be induced by the same 
methods used for the case of the flag algebra C[Mtti^_J^'« in the literature, for example, 
IGL961 iKlOSl IKM051 IMS051 [5tu95 j. See also [Vin95[ Theorem 1], for the properties of 
the algebra of polynomials on a semisimple algebraic group and its associated graded 
algebra. 

Theorem 3.8.1. The length k branching algebra B^^ for (GLra, GLn] is a flat de- 
formation of the Hibi algebra Ti^^ over C^^- 

Proof. Let us impose a filtration on B^ ^ by giving the following weight on each mono- 
mials. Fix an integer N greater than 2m, and then define the weight of I = [ii , • • • , Iq] G 



(3.8.1) wt(I) = ^irN^ 



r>1 

The weight of a standard tableau t consisting of 1^ is defined to be the sum of individual 
weights, i.e., wt(t) = ^j,wt(Ic). Then we can define a Z-filtration F^^ = {F^^} on 
B"^^^ = C[M^,k]^-^'^i- with respect to the weight wt. Set fT^^m,k'l t° space 
spanned by 

{At : wt(t) > d}. 

The filtration F^^ is well defined, since every product Yl ^ic expressed as a linear 

combination of standard monomials with bigger weights by Proposition 13.5.21 For all 
pairs A,B e C^^^, since wt(A) +wt(B) = wt(A AB) +wt(A VB), Sa^b and Saab Save 
belong to the same associated graded component. Therefore, we have sa -gr Sb = Saab 'gr 
savb where sc are elements corresponding to 5c in the associated graded ring gr^^[B^-^) 
of B^^ with respect to the filtration F^^. Then it is straightforward to show that the 
associated graded ring gr^^[B^-^^) forms the Hibi algebra over C^^. From a general 
property of the Rees algebras (e.g., |AB04] ), the Rees algebra TZ^ of B^y. with respect to 

pwt. 

d>0 

is flat over C[t] with its general fiber isomorphic to B^-^^ and special fiber isomorphic to 
the associated graded ring which is V.^^. □ 

We remark that Spec(?^{^^) is an affine toric variety in the sense of |Stu95| . Then, 
the rational polyhedral cone corresponding to the affine toric variety and the integral 
points therein can be realized from our description of the affine semigroup ^ given at 
the beginning of ^2.81 



4. Stable Range Branching Algebra for (Sp2Ta)Sp2n) 

In this section, starting from combinatorial descriptions of stable range branching 
rules, we study the affine semigroup algebra and its associated Hibi algebra for (Sp2^, Sp2n)- 
Then we construct an explicit model for the stable range branching algebra. Along with 
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these, we also show that these algebraic objects are isomorphic to their (GL2Ta, GL2n) 
counterparts with a proper length condition. 

Recall that we can label irreducible rational representation of Sp2ra! after identifying 
dominant weights with Young diagrams, by Young diagrams with less than or equal to 
m rows (cf. [GW09| §3.1.4]). We let T2^ denote the irreducible representation of Sp2Ta 
labeled by Young diagram F. 

4.1. Let Jra = (ja,b) be the m x ra matrix with ja,m+i-a = 1 for 1 < a < m and 
otherwise. Then we define the symplectic group S^2m. °^ rank m as the subgroup of 
Gl-2m preserving the skew symmetric bilinear form on C^^ induced by 



Jm 




Note that, for the elementary basis {e^} of the space C'^^, ej and e2m+^-j make an 
isotropic pair for 1 < j < m with respect to this bilinear form. Also, the subgroup of 
upper triangular matrices with 1 's on the diagonal can be taken as a maximal unipotent 
subgroup of Sp2Tn- We will denoted it by Usp^^^^- 

For n < m, we identify Sp2n with the subgroup of Sp2ra preserving the skew symmetric 
bilinear form restricted to the subspace of C^"^ spanned by 

{ea,e2m+i-Q : 1 < a < n}. 
Then Sp2n can be embedded in Sp2ra as follows. 



(4.1.1) 



X Y 
Z W 





" X 





Y " 









I 









. z 





w . 




is the 2fm 




n] X 2fm 



O's are the zero matrices of proper sizes. 

4.2. In order to construct an affine semigroup encoding stable range branching rules for 
(Sp2Ta> SP2n)i '^^ review the following combinatorial description of branching multiplici- 
ties. 



Lemma 4.2.1 ( [GW091 Theorem 8.1.5]). For Young diagrams F and D with < m 
and €(D) < m — 1, the multiplicity o/t^j^_^j in as a Sp2(Ta-i) representation 
is equal to the number of Young diagrams E satisfying the interlacing condition 
F □ E □ D. 

For example, if F = (5, 3, 3, 2, 1 ) and D = (4, 3, 2, 2], then the multiplicity of in t^q 
is equal to the number of E = (ei , 62, • • • , 65) in 

5 3 3 2 1 

ei 62 63 64 65 
4 3 2 2 



so that the entries are weakly decreasing from left to right along diagonals. 
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Note that this branching is not multiplicity free and rather similar to the two-step 
branchings for the general linear groups. To obtain a description of the multiplicity 
spaces for (Sp2^,Sp2^), we can simply iterate the above lemma. Because of the length 
condition £(Ei<) < k of S^jk representations r^]^ for n < k < m, it will be quite different 
from the (GL2Ta)GL2n) case (Proposition I3.2.T]) . Within the stable range £(F) < n, 
however, we have exactly the same description. 

In the previous example, if we set F = (5,3,3,2,0) so that £(F) = 4, then the multi- 
plicity of in x\q is equal to the number of E = (ei , ei, • • • ,65) in 

5 3 3 2 
ei ez 63 64 65 
4 3 2 2 

and the interlacing condition makes es = 0. Therefore the multiplicity of in t^q is 
equal to the multiplicity of the GLg representation in the GLio representation p^Q. 

Remark 4.2.2. (1) For complete GT patterns for 5^)2^) '^^ refer to |Kir88] and 
|Pr94| ■ See also |Ki08| for their ring theoretic interpretation. 
(2) The branching algebra for (Spim? Sp2m-2) without any restriction on the 
length of representations has interesting algebraic and combinatorial prop- 
erties with an extra structure from the action of SL2 x • • • x SL2. For this, 
we refer to |KYllj . 

4.3. Recall that 'P|^(F, D) is the set of all GT patterns for (GL2Ta)GL2n) whose types 
are F/D. Within the stable range £(F) < n, F ^ D implies £(D) < n, and therefore the 
support of every GT pattern in P^^IF, D) lies in the GT poset F^^^ of length n: 

^(2m) (2m) (2m) 

x,2 • • • s-n 

A2m-^) ^(2m-l) ^{2m-l) 

X.2 • • • x,,^ 

^(2n) (2n) (2n) 

A.^ A.2 A-n 

Proposition 4.3.1. Let F and D be Young diagrams with F I) D and £(F) < n. 
Then the branching multiplicity TTt(T^,T2^] is equal to the number of elements in 
'P|^(F, D), and therefore it is equal to the number of elements in 7^^(F, D). 

Proof. Prom Lemma l4.2.H by using the same argument used to prove (2) of Proposition 
[3:211 the set Vl^l^.^O) of GT patt erns of shape F/D counts the multiplicity of t^^ in 
T2^. The last statement follows from Proposition 12.8.1] □ 

We call the affine semigroup V'^-^, defined in ()2.7.ip . the semigroup for (Sp2TTi,) ^Vln) 
and call its associated semigroup algebra C['P|j^^] the semigroup algebra for (Sp2tti,) Sp2n) 
Then it is graded by A^^n defined in ^2.6[ 



C[^2m,n]= © C[P|-](p,o) 

(F,D)GAn,n 
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4.4. To define tableaux and standard monomials for the symplectic groups, we shall 
use the following ordered letters: 

(4.4.1) (2m) = {ui < vi < U2 < V2 < • • • < Utt^ < v^}. 

If we let C{lvn) denote the set of all non-empty subsets J of (2m), then on C{lvn) we 
can impose the tableau order ■<, as it is done in ^2.21 for Cim.) through the bijection 

(4.4.2) l(uc) = 2c - 1 and l(vc) = 2c 

for 1 < c < m. Then C{2vn) is a distributive lattice isomorphic to Ci^. 

For m > n, we consider the subposet £(n, 2m) of C{2vx) with all the elements J C 
(2m) of the forms 

(4.4.3) [ui,U2, ••• ,Uc,-yi,y2,--- ,ys], 

[U1,U2,--- ,Uc], 

where c < n and Un+i < yi < < • • • < Us ^ ^m- In particular, if G J for c < n, 
then {uh : 1 < H < c} C J. 

Now, for k < n, let /3(rL, 2m)ic be the subposet of /I(n, 2m) consisting of J G i2(rL, 2m) 
with IJI < k. Then, through the map (I4.4.2p . it is straightforward to see that £(n, 2m)ic 
is isomorphic to given in Definition 12.2. H and therefore isomorphic to ^ by 

Corollary [2321 

Definition 4.4.1. (1) The distributive lattice for (Sp2m)Sp2n) 

£sp = >C(rL,2m)n 

^ /-2n 

(2) The Hibi algebra for (Sp2m) Sp2n); denoted by Tisp, is the Hibi algebra over 
the distributive lattice Csp. 

Note that from £sp — ^zmw Hihi algebra Tisp for (Sp2Ta>Sp2n) is isomorphic to 
^2mn- Then from Proposition |2J£^ for (GL2ra, GL2 n) we have 

Corollary 4.4.2. The Hibi algebra for (Sp2m>Sp2n) ^5 isomorphic to the semigroup 
algebra for (Sp2m, Sp2n)- 

4.5. Next, we define standard tableaux for (Sp2m)Sp2n)- 

Definition 4.5.1. (1) A standard tableau t for (Sp2ra)Sp2n) ^5 cl multiple chain 
in £sp •■ 

t = (Ii ^•••^Is). 

(2) The shape slin(t) of a standard tableau t for (Sp2m,)Sp2n) is F/D where 
F = (|Ii|,--- andD = (di,--- ,dn) 

with dr being the number of uh 's in t for 1 < h < n. 
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We write 7sp(T^, D) for the set of all standard tableaux for [Spim^^Vzn) whose shapes 
are F/D, and consider the disjoint union 

rsp= U rsp(F,D) 

(F,D)eAn,n 

over An,n- Then as in the case of the general linear groups, 7sp gives rise to a C-basis for 
the Hibi algebra for (Sp2TTi) Sp2n)- -^s in §2.10| we shall identify monomials in the Hibi 
algebra Hsp with tableaux whose columns are elements of Csp- 

Proposition 4.5.2. (1) The Hibi algebra T-Lsp for (Spim) Spin) graded by An,n) 
and for each (F, D) € An,n; TsptF, D) forms a C-basis for the graded compo- 
nent ?^sp(F, D) ofUsv 
(2) The number of standard tableaux for (Spim? Sp2n) of shape F/D is equal to 
the branching multiplicity tti(t2^,T2^) o/t^^ in T2^. 

Proof. Prom the isomorphism £sp — ^imw easily see that there is a bijection 

between 7^p(F,D) and 7^^;^(F,D). Then (1) follows from Lemma [2302] and (2) follows 
from Proposition 14.3.11 □ 

4.6. We remark that every standard tableau for (Sp2m) Sp2rL) of shape F/D can be real- 
ized as a skew semistandard tableau of shape F/D having entries from {un+i , Vn+i , • ' ' > "^-m) ^ 
Por example, for m = 10 and n = 6, the standard tableau of shape F = (6,5,3,0,0) and 
D = (4,3,1) 

[U1,U2,U3] ^ [U1,U2,V4] ^ [ui,U2,V4] ^ [u, , U4] ^ [V4,U5] ^ [U5] 

in Csp = C{3, 10)3 can be identified with the skew semistandard tableau 











V4 


U5 








U4 








V4 


V4 









where the empty boxes in h-th row are considered as the ones with Uh for 1 < H < n. 

We also remark that, as it is shown in Proposition 12.3. 1[ we can attach an order 
increasing subset Ai of ^ to each I G £sp '■ 

(4.6.1) Ai= IJ a}'' 

2n<j<2m 

where a|'' C ^imn defined as 

■'^I ~ I 1 ' 2 ' ' Si f ) 

,(2i) _ f (2i) (2i) (2i)1 
M,j — A.^ > ^2 ' ' ti J • 

Here Si and tt are the numbers of elements in I less than or equal to ut and vi respectively. 
Then we can relate every element of 7sp to a sum of characteristic functions over these 
order increasing subsets as given in Proposition 12.8.11 and (12. 8. 111 . This gives a direct 
proof for Corollary 14.4.21 
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4.7. Now we want to lift the elements of the Hibi algebra "Hs-p to construct the sta- 
ble range branching algebra for (Sp2ra) Sp2n)- For this purpose, we briefly review the 
polynomial model of Sp2m^-representation spaces studied in |Ki08] . 

From ()3.4.ip . as a GL2ra module, C[M2ra,m]'^"^ decomposes into irreducible represen- 
tations — -^y taking Spim. ^ ^ subgroup of GL2ra, Sp2ra X GLra is acting 
on the space M2ra,m = C^"^ as the action of GL2Tn x GL^a given in (13.3. Ij) . 

Then we take the quotient of C[l\/l2ra,m]'^'^ by the ideal I$p = Y.i^^ where is the 



complement space to in i.e., g^^n 



• 1^ for each F (cf. [FH911 §17.3]). 



Then this quotient algebra can be taken as a polynomial model of the flag algebra for 
Sp2Ta ill that it contains exactly one copy of every irreducible representation '^\^- 

J'sv = C[M2n^,J^-/Xsp 

= L 

{(F)<m 

Moreover, this decomposition is compatible with the graded structure of the algebra, 
Tj' • C tiln^^^. Therefore, for the stable range £(F) < n, we can consider its 



I.e., 



subalgebra consisting of with £(F) < n: 



(4.7.1) 



Sp 



«(F1<rL 



'2m 



4.8. To describe generators of Jspi to each I = [wi, • • • , Wr] S C{2m.) with r < m, we 
attach a determinant function Sj/ as follows. For Q G M2m,m) let 6i/(Q) denote the 
determinant of the submatrix of Q = (ta^b) obtained by taking the i\,i2, ■ ■ ■ ,i^-th rows 
and the 1 , 2, • • • , r-th columns: 



(4.8.1) 



6i/(Q) =det 



ti'l tio 



tit 



where {1'^ , i-2> • • • > i-rl is the image of the set {wi , W2, • • • , w^} C (2m) under 
(4.8.2) i|; : {ui , vi , • • • , u^t^, Vt^^} ^ {1 , 2, • • • , 2m} 

\|){uc) = c and ij;(vc) = 2m + 1 — c 

for 1 < c < m. 

This conversion procedure is to make the labeling (u^Vc) of isotropic pairs, which are 
denoted by (c, c) in |Be86| and (2c — 1,2c) in [Ki08| . compatible with ours (c,2m+l — c) 
for the skew symmetric form defined in ^4.11 

Notation 4.8.1. To avoid a possible ambiguity, we impose a new total order < on 
{1,2, •• • ,2m} induced by i}) in '^'^d. the order of (2m) given in ^.4-l\} : 

1 < 2m < 2 < 2m -1 <-<m<m+1 

(1) To emphasize the order <, we shall use the prime symbol as in i- for the 
elements ij o/ {1 , 2, • • • , 2m}. 
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(2) Then, in the determinant Iji4-8.1\ ), we may further assume that 



i\ < i2 < • • • < ij. 



to fix the sign of the determinant. 
(3) We also let I' denote the image o/ I € £(2m) under \\). Similarly, we let t' 
denote the multiple chain (I{ ^ I2 ^ • • • ^ Ic) corresponding to the multiple 
chain t = (Ii ^ I2 ^ • • • ^ Ic) C{2m). 



For the flag algebra Tsp, we are interested in 5i/ with I G £(2m) whose h-th smallest 
entry is not less than uh for all h > 0. 

Definition 4.8.2 ( [Be86t [Ki08] ). Fix the element Jo = [ui,U2,--- ,Ura] € £(2m) of 
length m. For a multiple chain t = (Ii ^ I2 ^ ■ ■ ■ ^ Ic) of C{lvn) , its associated 
monomial 

is called a -standard monomial, if Ig >z Jo for all s. 



4.9. The ideal Xsp is finitely generated. Using the elements of Zsp (cf. [FH91t §17.3]) 
combined with standard monomial theory of C[M2ra,m]^"^i [Ki08j shows that Sp-standard 
monomials project to C-basis elements of the quotient algebra Tsp, and that they are 
compatible with the graded structure of the algebra. 

To a product of 5i/'s, as an element of C[M2m,m]^'^i a-PPly the straightening relations 
in Proposition 13.5.2] to obtain a linear combination of standard monomials for GL2Ta: 

i r j>l 

If there is a non-zero term Y[j ^k' which is not a Sp-standard monomial, then apply 
relations from the ideal Zsp, which replace the entries in K^j's corresponding to isotropic 
pairs (Ua,Va) with the sum of entries corresponding to (ut,,Vb) for a < b, thereby ex- 
pressing Yij ^K' as a linear combination of Sp-standard monomials. For further details, 
we refer to |Ki08] . A combinatorial description of this procedure in the language of 
tableaux is given in |Be86| . 

Proposition 4.9.1 ( [Ki081 Theorem 5.20]). Sp-standard monomials project to a C- 
basis of the flag algebra J"sp /o?" ^Vim- ^'^ particular, for a Young diagram F with 
£(F) < m, Sp-standard monomials of shape F project to a weight basis for the Spim, 
irreducible representation T2^ C J-sp . 

We also note that, from the graded structure '^2^ ' '^im ^ '^'im'^ °^ -^Sp' order to 
obtain the subalgebra in (j4.7.ip . it is enough to consider 5i/'s with I G >C(2ra) and 
HI < n. 
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4.10. From (jl.S.lj) and ()4.7.ip . we want to find an explicit model for the Usp^^ -invariant 
subalgebra of T^^^ : 



is graded by An,n/ CLiT-d for each (F, D) € An,n the Sp standard monomials At' cor- 
responding to standard tableaux t for (Sp2ra) ^Vin) 'whose shapes are F/D form a 
C-basis of the (F, D] -graded component. The dimension of the (F, D) -graded compo- 
nent is equal to the branching multiplicity o/t^^ in T2^. 

Proof. Recall that, for I € >Csp C C{2m), we defined the polynomial 5i/ on the space 
M2m,m in (I4.8.1P . By ()4.4.3p and ()4.8.2p , it is the determinant of a submatrix of Q G 
M2m,m obtained by taking consecutive columns {1,2, •• • , |I|} and either consecutive rows 
{1 , 2, • • • , r} or partially consecutive rows {1 , 2, • • • , r} U {bi , • • • , bj} or only {bi , • • • , bj} 
of Q for r < n and b^ € {n -|- 1 , n -|- 2, • • • , 2m — n} for all i. 

Since the left action of Uin C GL2Ta under the embedding (j4.1.ip operates the rows of 
M2ra,mi the elements Sji for I G £sp ^^e invariant under the action of U2n and therefor 
invariant under the action of Ugp^^- Since the ideal Xsp is stable under the action of 
Sp2mi the generators of the algebra Bsp are invariant under the unipotent subgroup 
llsp2^ of Sp2n) and so are their products. Also, since every I £ £sp satisfies |I| < n, we 



On the other hand, every element in £sp is greater than Jo = [ui,U2,--- ,Ura] with 
respect to the tableau order, and therefore standard tableaux t for (Sp2Tn) Sp2n) (Defi- 
nition 14.5.11 and Proposition 14.9. ip give rise to Sp-standard monomials At' (Definition 
14.8. 2p for Tsp. That is, Sp-standard monomials corresponding to standard tableaux for 
(Sp2m>Sp2n) project to linearly independent elements in the Usp^^ -invariant subalgebra 
of Jj^' C Jsp- They span the whole Usp2^ -invariant subalgebra of -T^sp', because for each 
(F, D) G An,,n the number of standard tableaux in 7sp(F, D) is equal to the multiplicity 
of in by Proposition 14.3.11 Furthermore, they are scaled by weight D under 
the action of the diagonal subgroup {dlag(ai , • • • , Qn, a^^ , • • • , )^ °^ ^Vin as given in 
()3.7.ip . Therefore, standard monomials At' with t G 7sp(F, D) are the highest weight 

vectors of the copies of t^^ in ^2^. This shows that Bsp = ^-^sp') ''^'^ ^^"^ it^ graded 
structure. □ 

In this sense, we call Bsp the stable range branching algebra for (Sp2TTt) Sp2n)- Recall 
that we obtained 8$^ by lifting the elements of the Hibi algebra ^sp o'^er the distributive 
lattice £sp which is isomorphic to the distributive lattice n- Now we compare it with 




«(F)<Tl 



In 



«(F)<Tl D 

Theorem 4.10.1. The subalgebra Bsp of Tsp generated by 

{5i'+Xsp :le/:sp} 
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the algebra B^^^ (Definition l3.6.ip obtained from the Hibi algebra T-t^^ for the general 
linear groups. 

Proposition 4.10.2. The stable range branching algebra Bs-p for (Sp2ra)Sp2n) 
isomorphic to the length n branching algebra B^^ for (GL2Ta) GL2n] • 

Proof. Prom the isomorphism Cs-p = -^Imn °^ distributive lattices, with I i— > I, we can 
consider a bijection between the generating set of B$-p and the generating set of B^^^: 

[bv +Xsp : I G Ap} ^ {5t : t G C^^^] 

Then, to see that this bijection gives rise to an algebra isomorphism, let us show that the 
straightening relations among 8j-'s in B^^ agree with those of (5i/ +Tsp)'s in Bsp C Fs-p- 
As explained in §4.9| to express a product of 5i''s as a linear combination of Sp- 
standard monomials projecting to the quotient Jsp = C[M2m,m]^"V^Sp) '^^ first apply 
the straightening relations in C[M2Ta,m]^"^ (Proposition I3.5.2]) and then relations from 
the ideal Xsp . 

Por elements € >Csp C C{lm), the corresponding product Oi^i- ^ element in 
C[M2m,m]'^''^ can be expressed as a linear combination of GL2Ta standard monomials: 

(4.10.1) n^'=Z'^^n^'<;,i 

i T j>l 

in C[M2Tu,m]^''^- Now we claim that for each non-zero term Oj^K'.i indexes Kt-j's 
form a multiple chain in >Csp, i-e-, the monomial Oj ^K' . is already Sp-standard, and 
therefore the expression (I4.10.ip provides the Sp-standard monomial expression of Yii ^i.' 
projecting to Ssp C /"sp- This follows directly from the quadratic relation ()3.5.2p . that 
is, for I, J e £sp, 

6i,5j/ = ^ Ct5s;5t;. 

r 

On the right hand side, for each non-zero term 5s; Sj,') the chain Sr ^ Tr satisfies the 
condition Sr ^ Jo and Tr ^ Jo in Definition I4.8.2[ which can be easily seen from the 
statement (2) of Corollary 13.5.41 and the fact that I and J from £sp do not contain Vh for 
1 < h < n. 

Moreover, by Theorem 14.10.11 and Proposition 14.3.11 each (F, D) homogeneous spaces 
of both algebras are of the same dimension, and they have C-bases labeled by the same 
patterns. Therefore, two graded algebras are isomorphic to each other. □ 

With this characterization Ssp = B^^, from Theorem 13. 8. 1[ we have 

Corollary 4.10.3. The stable range branching algebra Bsp for (Sp2m,) Sp2Tv) a fiat 
deformation of the Hibi algebra T^sp for (Sp2Ta) Sp2n)) which is isomorphic to T-Li^^- 
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5. Stable Range Branching Algebra for (S0p,S0q) 
Through out this section, for m > n > 2, we set 

p = 2ra + 1 or 2m; 
q = 2n + 1 or 2n; 
k = n if q = 2n + 1 , 
= n — 1 if q = 2n. 

Following the same techniques we developed for the symplectic groups, we construct the 
stable range branching algebra Bso for (S0p,S0q). The results and their proofs in this 
section are analogous to the case of [Spim^^Vzn)- 

5.1. Let us review a labeling system for the irreducible rational representations of SOp 
(cf. [GW091 §3.1.4]). For the even orthogonal group Ozm. of rank m, every Young 
diagram F with £(F) < m can label exactly one irreducible representation Cim.' '^hich can 
be also realized as a SOim irreducible representation. The missing ones for S02ra are the 
pairs of associated irreducible representations and appearing as the components 
of irreducible representations o'2^ of 02m labeled by Young diagrams F with m rows, 



I.e., a 



2m 



^2m 



CT2m- o^*^ special orthogonal group S02m+i of rank m, every 

irreducible rational representation o"2^^^ can be uniquely labeled by a Young diagram F 
with £(F) < m. Then these representations are also 02m+i "irreducible. 

5.2. Let Jm = (jQ,b) be the m x m matrix such that ja,m+i-a = 1 for 1 < a < m 
and otherwise. Then we define the special orthogonal groups S02m and S02m+i as 
the subgroups of SL2m and SL2m+l preserving the symmetric bilinear forms on C^™ and 
(j~.2m+i induced by 

, Jr 

and 





Jm 



Jm 







Jm 



1 











respectively where O's are the zero matrices of proper sizes. Then, the pairs (ej,ep+i_j) 
of the elementary basis elements for make isotropic pairs with respect to the above 
symmetric bilinear form. Also, the subgroup of upper triangular matrices with 1 's on 
the diagonal can be taken as a maximal unipotent subgroup of SOp. We will denote it 
by Usop • 

For m > n, let us identify S02n as the subgroup of SOp preserving the symmetric 
bilinear form on the subspace of spanned by {ej, Ep+i-j : 1 < j < tl}- Then we can 
embed S02n in SOp as follows 

X Y 



X 
Z 



Y 
W 





z 





w 



where X, Y, Z, W are blocks of size n x n, I is the (p — 2n] x (p — In) identity matrix, 
and O's are the zero matrices of proper sizes. Similarly, we embed S02n+i in S02m+i by 
considering the (In + 1 ) -dimensional subspace of C^^'^^ spanned by {ei,e2m+2-i : 1 < 
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a < n} and e^+i. For S02n+i in S02rai '^^ ^se the (2tl + 1 ) -dimensional subspace of 
C^"^ spanned by {ej, 62^+1-5 : 1 < j < n} and [e^ + e^^+i ). 

5.3. Our next task is to construct an affine semigroup encoding stable range branching 
rules for (SOp, SOq). Note that (fi, • • • , fm) £ is a dominant weight for S02m+i and 
S02m) if fl > ■ ■ ■ > f m ^ and fi > • • • > fm-i > |fml > respectively. 

Lemma 5.3.1 f |GW09[ Theorems 8.1.3 and 8.1.4]). (1) Let f = ^fn,) and 

D = (di,--- , dra) be dominant weights for S02m+i '^^'^ S02m respectively. 
Then the branching multiplicity of ^2-^^ in o'2^^j is equal to 1 if [f], - ■ ■ ,fm] 
interlaces (di,--- ,|dral), i-e., 

fi h ■■■ fm-i fm 

di d2 ••• d^u-i Idml 

and otherwise; 

(2) Let F = (f 1 , • • • , fra) and D = (di , • • • , dra-i ) be dominant weights for SO2 



m 



and S02m-i respectively. Then the branching multiplicity o/o'2Ta-i "-'2m 
equal to 1 i/ (fi, • ' ' > l"fml) interlaces (di, • • • , dra), i-S-, 

fl f2 ■ ■ ■ fra-l Ifral 

di d2 ■ ■ ■ dm-i 

and otherwise. 

By iterating these results, we may obtain patterns counting the branching multiplic- 
ities for (SOp, SOq]. Such patterns are different from the GT patterns for (GLp,GLq]. 
Within the stable range, however, they are the same as the ones for (GLp,GLq) with 
restrictions on lengths. That is because, as in the case for the symplectic groups, the 
length restriction £(F) < k forces £(D) < k via the interlacing conditions in Lemma [5. 3.11 
Therefore, as is shown in Proposition 14.3.11 for the symplectic groups, we have 

Proposition 5.3.2. Let F and D be Young diagrams with F I) D and £(F) < k. Then 
the branching multiplicity m(o'^, is equal to the number of elements in "Pp (F, D), 
and therefore it is equal to the number of elements in Tp[y,D). 

As in the case of (GLp, GLq) in ()2.7.ip . we can consider the afSne semigroup V^i^ of 
the order preserving maps from the GT poset F^^ of length k: 

X, X2 x^. 

^(p-i) ^(P-I) ^{p-D 

X, X2 • • • 

to non-negative integers. We call "P^^ the semigroup for (SOp, SOq), and define its 
associated semigroup algebra: 

C[^pV]= © C[Pp^](,o) 

(F,D)GAk,k 
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and call it the semigroup algebra for (S0p,S0c 



5.4. Let us define the distributive lattice for (S0p,S0q) and study its Hibi algebra. 
We shall closely follow the construction developed in ^4.41 for the symplectic groups. 
Consider the ordered letters: 

(5.4.1) (2m) = {ui < vi < U2 < V2 • • • < < v^} , 

(2ra + 1 ) = {Ui < Vi < U2 < V2 • • • < Um < Vra < oo} 

for p = 2ra and 2ra + 1 respectively. 

If we let C{p) denote the set of all non-empty subsets J of (p), then on £(p) we can 
also impose the tableau order ^ as in ^2.21 and ^4.4[ Then C{p) is a distributive lattice 
isomorphic to C-p, as in the case of the symplectic groups, through the bijection ()4.4.2j) 
(and l(oo) = 2m +1 for p = 2m + 1 ). 

Then, we define C{n, q,p) to be the set of nonempty subsets J of C{p) of the following 
forms: 

(5.4.2) [ui,U2,--- ,Uc,yi,y2,--- ^Vs], 

[ui,U2,--- ,Uc], 

where c < n and, for q = 2rL and 2rL + 1 , 

Un+i < yi < yi < • • • < ys; 
vn+i < yi < y2 < • • • < ys- 

respectively. In particular, if Uc G J for c < n, then {u^ : 1 < H < c} C J. 

Now, let £(n, q,p)k be the subset of £(n, q,p) consisting of J with |J| < k. Then, as is 
the case for the symplectic groups ( g4.4p . we can identify i2(rL, q,p)k with the distributive 
lattice C^_^^^^, and therefore with C^^ by Corollary 12.3.21 

Definition 5.4.1. The distributive lattice for (S0p,S0q) is /I(rL, q,p)k, and it will 
be denoted by Cso- 

£so = >C(n, q,p)ic 

Then we define the Hibi algebra for (S0p,S0q), denoted by Hso, to be the Hibi 
algebra over the distributive lattice £so- Prom the isomorphism of distributive lattices, 
we have Tiso — ^pk- Then from Proposition 12. 10.3) for (GLp, GLq), we have 

Corollary 5.4.2. There is an algebra isomorphism 



Uso = C[Pp\] 
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5.5. As in the previous cases ( ^2.10p . we shall identify the monomials in the Hibi algebra 
T-Lso with tableaux whose columns are elements of £50- 

Definition 5.5.1. A standard tableau t for (SOp, SOq) is a multiple chain Ii ^ • • • ^ 
Is in Cso ■ '^he shape sh.n(t) o/t is F/D where F = (|Ii |, • • • , |Isl)^ and D = (di , • • • , dn) 
with dr being the number ofur's in t for 1 < r < n. 

We write 7so(T^)D) for the set of all standard tableaux for (SOp, SOq) whose shapes 
are F/D, and set 

Tso= U rso(F,D) 

(F,D)eAk,k 

Then as in the case of the symplectic groups, 7so gives rise to a C-basis for the Hibi 
algebra for (SOp, SOq). 

Proposition 5.5.2. (1) The Hibi algebra Tiso /o'" (SOp, SOq) is graded by Aj^)^ 
and 7so(F, D) forms a C-basis of the graded component T-Lsoi^-,^) ■ 
(2) For (F, D) € Fi^^^, the number of standard tableaux for (SOp, SOq) of shape 
F/D is equal to the branching multiplicity m(a^,CTp) of in o^. 

Proof. Prom the isomorphism £50 — ^p k' straightforward to see that there is a 
bijection between 7so(T^)D) and Tp[¥,D). Then (1) follows from Lemma [2.10.21 and (2) 
follows from Proposition 15.3.21 □ 

5.6. We can also find a correspondence between Cso and the set of order increasing 
subsets of the GT poset ^ in the same way explained in §4.61 Namely, define the order 
increasing subset Ai of F^j, corresponding to I € Cso as 

(5.6.1) Ai= U {xS'',xi'V--,x«'} 

q<j<p 

where, for n + 1 < h < m, S2h-i and S2h are the numbers of elements in I less than or 
equal to and respectively; and szn is the number of elements in I less than Vn and 
S2m+l is the number of elements in I. Then every element of 7so can be related to a sum 
of characteristic functions over these order increasing subsets as given in Proposition 
EXHand ()2.8.111 . This gives a direct proof for Corollary [5A2l 

5.7. To construct the stable range branching algebra for (SOp, SOq), we review the 
polynomial model of SOp-representation spaces studied in [Ki09j . 

Prom (13. 4. 111 . C[Mp^Tu]^'^ consists of GLp-irreducible representations with £(F) < m. 
By taking Op as a subgroup of GLp, Op x GLra is acting on the space Mp^ra — i^iC^ via 
the action of GLp x GL^a given in (j3.3.ip . Then we take the quotient of C[Mp^ttl]^"^ by the 
ideal Zq = where is the complement space to the Op -irreducible representation 

al in p^, i.e., p^ = cr^ eX"" for each F (cL [PH9H §19.5]). 

Then [Kl09] shows that this quotient algebra can be taken as a polynomial model 
for the flag algebra for SOp in that it contains exactly one copy of each irreducible 
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representation with £(F) < m: 

-Fso = C[Mp,J^-/Xo 

= L 

£(F)<m 

and it is graded by Young diagrams, i.e., ap' • C ap'^^^. We note that (y^^ with 
£(F) = m are irreducible Ozm. representations, but they are not irreducible as SOim 
representations. 

To take the stable range £(F) < k, we consider its subalgebra consisting of o"p with 
< k: 



(5.7.1) 



r 



so 



(FXk 



5.8. To describe generators of JsOi to each I = [wi,--- ,Wr] G >C(p), we attach a 
determinant function 5i/ as follows. 

For Q G Mp^Tui 'we let 6i'(Q) denote the determinant of the submatrix of Q = (ta_b) 
obtained by taking the , • • • , i^-th rows and the 1 , 2, • • • , r-th columns: 



(5.8.1) 



6i/(Q) =det 



"ti'i "ti'i 



ti',r 



ti' 



where is (1'^ ) i-2) ' ' ' > ^ri image of the set {wi , W2, • • • , w^} C (p) under \|;p: 

(5.8.2) il;2m : {u-i , vi , • • • ,Utt^,v^} — >{1,2, ••• ,2m} 

il^2m(u-c) = c and i|j2m(vc) = 2m + 1 - c; 
, V-| , • • • , Uta, Vrn., oo} — > {1 , 2, • • • , 2m, 2m + 1 } 
il^zm+itu-c) = c and \|j2m+i (vc) = 2m + 2 - c 

for p = 2m and 2m + 1 respectively, for 1 < c < m and i|^2m+i (oo) = m + 1 . 

Then, with the bijection \|;p, we can impose a new order < on {1 , 2, • • • , p} induced by 
the order on (p) in (j5.4.ip : 

1 < 2m < 2 < 2m — 1 <---<m<m+l; 

1 < 2m + 1 < 2 < 2m <---<m<m + 2<m + l 

and we keep using the convention of I', 5i/ and At' used for the symplectic groups 
(Notation 14.8. Ij) . This conversion procedure is to make our labeling [uc,Vc) of isotropic 
pairs ( ^5:2]) compatible with those used in |KW931 [Kl09] . 

To I = [wi,--- ,Ws] G ^(p), we attach a determinant function hy as we define in 
()5.8.ip . For a multiple chain t = (Ii ^ • • • ^ It) of i2(p), let t(a, b) denote the a-th 
smallest element in the b-th column I^, of the tableau t. Also, let oi^c and (32c be the 
numbers of elements less than or equal to in Ii and I2 respectively. 
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Definition 5.8.1 (cf. [KW931 rPr94| ). Then the corresponding monomial 

is called an 0-standard monomial, if, in the chain t = (Ii • • • ^ It), 

(1) (X2c + Pic ^ 20 for 1 < c < m, and 

(2) if (X2c + Pic = 20 for some c with t(a2c) 1 ) = iJ-c t(P2c) = Vc for some b, 
t/ien t(p2c ~ 1 > b) = Uc. 

In |KW93| and |Pr94| . the above conditions (1) and (2) are used to define Young 
tableaux describing weight basis elements of irreducible Op representations. 

5.9. The ideal Xq is finitely generated. Using the elements of Iq (cf. |FH9H §19.5]) 
combined with standard monomial theory of C[Mp^ra]'^'^, |KiQ9] shows that 0-standard 
monomials project to C-basis elements of the quotient algebra J^sOi and that they are 
compatible with the graded structure of the algebra. 

To a product of 5i/'s in C[Mp^TTi]^"\ we apply the straightening relations in Proposition 
l3.5.2l to obtain a linear combination of standard monomials for GLp: 

i r j>l 

If there is a non-zero term Yi^ ^k' which is not an 0-standard tableau, then apply 
relations from the ideal Iq, which replace the entries of K^j's corresponding to isotropic 
pairs [uayVa] with the sum of pairs (ub,Vb)'s (and (oo,oo) for p = 2m + 1) for some 
a < b, thereby expressing Yij ^k' ^ ^ linear combination of 0-standard monomials. 
For further details, we refer to |KiQ9] . A combinatorial description this straightening 
procedure in the language of tableaux is given in |KW93] . 

The following is shown in [KIQ9] . See also |KW93| and [Pr94] . 

Proposition 5.9.1 ( |KiQ91 Theorem 3. 6, Proposition 3.9]). 0-standard monomials project 
to a C-basis of the flag algebra Tso for SOp. In particular, for a Young diagram 
? with £(F) < m, 0-standard monomials of shape F form a weight basis for the 
Op -irreducible representation ct^ C .7-so- 

5.10. Our next task is, from the discussions ()1.3.1|) and (I5.7.ip . to find an explicit model 
for the Us Oq -invariant subalgebra of J^^q- 

«(F)<k 
«(F)<k D 

Theorem 5.10.1. The subalgebra Bso of Tso generated by 

{6i, +Xso:Ie/:so} 

is graded by An.,n; and for each (F, D) G Ai^^i^ the O standard monomials A^/ corre- 
sponding to standard tableaux t for (SOp, SOq) whose shapes are F/D form a C-basis 
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of the [¥,D)-graded component. The dimension of the [¥,D)-graded component is 
equal to the branching multiplicity of in . 

Proof. For I G Cso C jC{'p), we defined the polynomial Sj/ on the space Mp^T^i iii ()5.8.1I) . 
By (I5.4.2P and and ()5.8.2j) . it is the determinant of a submatrix of Q € M2m,m obtained 
by taking consecutive columns {1,2, •• • , |I|}, and either consecutive rows {1,2, •• • ,r} or 
partially consecutive rows {1 , 2, • • • , r} U {bi , • • • , bg} or only {bi , • • • , bj} of Q for r < n 
and bi, G {n + 1 , n + 2, • • • , p — n}. 

Since the left action of Uq C GLp, under the embedding given in §5.2| operates the 
rows of Mp^iTi, all the determinants Sj/ for I € Cso ^^e invariant under the action of 
Uq, and therefor invariant under the action of Usoq- Since the ideal Zq is stable under 
the action of Op, the generators of the algebra Bso a-^e invariant under the unipotent 
subgroup UsOq of SOq, and so are their products. Also, since every I £ Cso satisfies 

III < k, we have Bso Q (-^sS)^'°' ■ 

On the other hand, for every chain I ^ J in Cso, ^I'^j' satisfies the conditions (1) and 
(2) in Definition 15.8. 1[ which can be easily seen from the statement (3) of Corollarv 13.5.41 
and the fact that I and J from £so do not contain Vh for 1 < h < n. This implies that 
standard monomials At' corresponding to standard tableaux t for (S0p,S0q) project to 
linearly independent elements in the UsOq -invariant subalgebra of C /"so- They 
span the whole UsOq -invariant subalgebra of J^^^, because for each (F, D) G A^^i^ the 
number of standard tableaux in Tsoi^i^) is equal to the multiplicity of in by 
Proposition 15.3.21 Furthermore, they are scaled by weight D under the action of the di- 
agonal subgroup {dtag(ai,--- , an, a;^\ • • • ,a7^)} or {diag(ai,--- ,an,^,a^\■■■ ,af^)} 
of SOq. Therefore, standard monomials A^i with t € Tsoi^y^) are the highest weight 

vectors of the copies of Tq in Tp. This shows that Bso = (-^so ) ^^'^ graded 
structure. □ 

In this sense, we call Bso the stable range branching algebra for (S0p,S0q). Recall 
that we obtained Bso by lifting the elements of the Hibi algebra ^so over the distributive 
lattice Cso which is isomorphic to the distributive lattice C^^. Now we compare it with 
the algebra B^ ^ (Definition 13.6. ip obtained from the Hibi algebra ^ for the general 
linear groups. 

Proposition 5.10.2. The stable range branching algebra Bso for [S0p,S0q) is iso- 
morphic to the length k branching algebra for (GLp, GLq). 

Proof. From the isomorphism £so — ^^Iv °f distributive lattices, with I i— > I, we can 

p,K 

consider a bijection between the generating set of Bso and the generating set of B^^: 

[Sy + Jo : I G Ao) ^ {St : t G C^^} 

Then, to see that this bijection gives rise to an algebra isomorphism, let us show that the 
straightening relations among 5<j-'s in B^y_ agree with those of (8i' +Zo)'s in Bso C J^so- 
As explained in ^5.9[ to express a product of 6i/'s as a linear combination of O- 
standard monomials projecting to the quotient Tso = 'C[Mp^rn}^'^ /^o, we first apply 
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the straightening relations in C[Mp in^]^"^ (Proposition 13.5.2]) and then relations from the 
ideal Xq. 

A product of representatives Oi^i'' element in C[Mp^TTi^]^'^ can be expressed as 

a linear combination of GLp standard monomials: 

(5.10.1) n^it'=L'^^n^K;, 

i T j>l 

inC[Mp,J^-. 

Now we claim that for each non-zero term Y[] ^k' > the indexes Krj's form a multiple 
chain in >CsOi therefore ()5.10.1j) gives 0-standard monomial expression of Yii ^i.' project- 
ing to Sso C Tso- This follows directly from the quadratic relation (j3.5.2p . For every 
chain I ^ J in Cso, byby satisfies the conditions (1) and (2) in Definition 15.8. 1[ which 
can be easily seen from the statement (3) of Corollary 13.5.41 and the fact that I and J 
from Cso do not contain for 1 < h < n. 

Moreover, from Theorem l5.10.1l and Proposition l5.3.2l each (F, D) homogeneous spaces 
of both algebras are of the same dimension with bases labeled by the same patterns. This 
shows that two graded algebras are isomorphic to each other. □ 

With this characterization B^o = B^-^, from Theorem 13. 8. 1[ we have 

Corollary 5.10.3. The stable range branching algebra Bso for (S0p,S0q) is a flat 
deformation of the Hibi algebra Hso for (S0p,S0q), which is isomorphic to T-L^^. 

Acknowledgment. The author thanks Roger Howe and Steven Glenn Jackson for in- 
sightful conversations regarding several aspects of this work. 
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